On Herding in Deep Networks

Maximum likelihood learning in Markov Random Fields (MRFs) with multiple layers of hidden units is typically performed using contrastive divergence or one of its variants. After learning, samples from the model are generally used to estimate expectations under the model distribution. Recently, Welling proposed a new approach to working with MRFs with a single layer of hidden units. The approach, called herding, tries to combine the two stages, learning and sampling, into a single stage. Herding runs the network as a deterministic dynamical system, similar to a Hopfield network. Expectations computed over trajectories of the dynamical system can be shown to converge to expectations computed over the data. In this technical report, we investigate herding in MRFs with multiple layers of hidden units, so-called deep networks. We derive the herding dynamics for deep networks, and we investigate the effect of three important characteristics on the performance of classifiers that are trained on energy averages over the herding trajectories: (1) the effect of the step size employed in the herder, (2) the effect of different initializations of the network in the positive and in the negative phase, and (3) the effect of different network architectures. From the results of our experiments, we observe that, although in networks with a single layer of hidden units the performance can be proven to be equal for finite step sizes, the step size highly influences the performance of herders in deep networks. Moreover, our results suggest that herding in deep networks requires a different type of network architecture than deep MRF models that are trained using maximum likelihood learning. Presumably, these experimental observations can be explained by a decoupling phenomenon in the top hidden units: the top hidden units run decoupled from the data, as a result they only feed noise into the network. In order to successfully herd in deep networks, future work should develop and investigate approaches to prevent higher hidden layers from decoupling. On Herding in Deep Networks Laurens van der Maaten ICT Group, Delft University of Technology